{
    "cells": [
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "<i>Copyright (c) Recommenders contributors.<br>\n",
                "Licensed under the MIT License.</i>\n",
                "<br>\n",
                "# Wide-and-Deep Model Hyperparameter Tuning with AzureML"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "This notebook shows how to auto-tune hyperparameters of a recommender model by utilizing **Azure Machine Learning service** ([AzureML](https://azure.microsoft.com/en-us/services/machine-learning-service/))<sup><a href=\"#azureml-search\">a</a>, <a href=\"#azure-subscription\">b</a></sup>.\n",
                "\n",
                "We present an overall process of utilizing AzureML, specifically [**Hyperdrive**](https://docs.microsoft.com/en-us/python/api/azureml-train-core/azureml.train.hyperdrive?view=azure-ml-py) component, for the hyperparameter tuning by demonstrating key steps:\n",
                "1. Configure AzureML Workspace\n",
                "2. Create Remote Compute Target (GPU cluster)\n",
                "3. Prepare Data\n",
                "4. Prepare Training Scripts\n",
                "5. Setup and Run Hyperdrive Experiment\n",
                "6. Model Import, Re-train and Test\n",
                "\n",
                "In this notebook, we use [**Wide-and-Deep model**](https://ai.googleblog.com/2016/06/wide-deep-learning-better-together-with.html) from **TensorFlow high-level Estimator API (v1.12 or higher)** on the movie recommendation scenario. Wide-and-Deep learning jointly trains wide linear model and deep neural networks (DNN) to combine the benefits of memorization and generalization for recommender systems.\n",
                "\n",
                "For more details about the **Wide-and-Deep** model:\n",
                "* [Wide-and-Deep Quickstart notebook](../00_quick_start/wide_deep_movielens.ipynb)\n",
                "* [Original paper](https://arxiv.org/abs/1606.07792)\n",
                "* [TensorFlow API doc](https://www.tensorflow.org/api_docs/python/tf/estimator/DNNLinearCombinedRegressor)\n",
                "  \n",
                "Regarding **AuzreML**, please refer:\n",
                "* [Quickstart notebook](https://docs.microsoft.com/en-us/azure/machine-learning/service/quickstart-create-workspace-with-python)\n",
                "* [Hyperdrive](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-tune-hyperparameters)\n",
                "* [Tensorflow model tuning with Hyperdrive](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-train-tensorflow)\n",
                "\n",
                "---\n",
                "<sub><span id=\"azureml-search\">a. To use AzureML, you will need an Azure subscription.</span><br>\n",
                "<span id=\"azure-subscription\">b. When you web-search \"Azure Machine Learning\", you will most likely to see mixed results of Azure Machine Learning (AzureML) and Azure Machine Learning **Studio**. Please note they are different services where AzureML's focuses are on ML model management, tracking and hyperparameter tuning, while the [ML Studio](https://studio.azureml.net/)'s is to provide a high-level tool for 'easy-to-use' experience of ML designing and experimentation based on GUI.</span></sub>"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 1,
            "metadata": {},
            "outputs": [],
            "source": [
                "%reload_ext autoreload\n",
                "%autoreload 2"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 2,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "Azure ML SDK Version: 1.0.10\n",
                        "Tensorflow Version: 1.12.0\n"
                    ]
                }
            ],
            "source": [
                "import os\n",
                "import itertools\n",
                "import shutil\n",
                "from tempfile import TemporaryDirectory\n",
                "\n",
                "from IPython.display import clear_output\n",
                "import numpy as np\n",
                "import pandas as pd\n",
                "import sklearn.preprocessing\n",
                "import tensorflow as tf\n",
                "\n",
                "import azureml as aml\n",
                "import azureml.widgets as widgets\n",
                "import azureml.train.hyperdrive as hd\n",
                "\n",
                "from recommenders.utils.timer import Timer\n",
                "from recommenders.utils.constants import SEED\n",
                "from recommenders.utils.tf_utils import pandas_input_fn_for_saved_model\n",
                "from recommenders.datasets import movielens\n",
                "from recommenders.datasets.pandas_df_utils import user_item_pairs\n",
                "from recommenders.datasets.python_splitters import python_random_split\n",
                "import recommenders.evaluation.python_evaluation as evaluator\n",
                "\n",
                "print(\"Azure ML SDK Version:\", aml.core.VERSION)\n",
                "print(\"Tensorflow Version:\", tf.__version__)\n",
                "\n",
                "# Temp dir to cache temporal files while running this notebook\n",
                "tmp_dir = TemporaryDirectory()"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "### 1. Create and Configure AzureML Workspace\n",
                "**AzureML workspace** is a foundational block in the cloud that you use to experiment, train, and deploy machine learning models via AzureML service. In this notebook, we 1) create a workspace from [**Azure portal**](https://portal.azure.com) and 2) configure from this notebook.\n",
                "\n",
                "You can find more details about the setup and configure processes from the following links:\n",
                "* [Quickstart with Azure portal](https://docs.microsoft.com/en-us/azure/machine-learning/service/quickstart-get-started)\n",
                "* [Quickstart with Python SDK](https://docs.microsoft.com/en-us/azure/machine-learning/service/quickstart-create-workspace-with-python)\n",
                "\n",
                "There are several ways to create an Azure Machine Learning service workspace.\n",
                "* Option 1: Use Azure portal\n",
                "    1. Sign in to the [Azure portal](https://portal.azure.com) by using the credentials for the Azure subscription you use.\n",
                "    2. Select **Create a resource** menu, search for **Machine Learning service workspace**, and select **Create** button.\n",
                "    3. In the **ML service workspace** pane, configure your workspace with entering the *workspace name* and *resource group* (or **create new** resource group if you don't have one already), and select **Create**. It can take a few moments to create the workspace.\n",
                "    4. Download **config.json** file from the portal's AzureML workspace page and place it to `<this-notebook-folder>/aml_config/config.json`\n",
                "* Option 2: Use [AzureML SDK](https://docs.microsoft.com/en-us/python/api/overview/azure/ml/intro?view=azure-ml-py#workspace) - Run following cell\n",
                "    * To find the full list of supported region, use Azure CLI from [your machine](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest) or [cloud shell](https://azure.microsoft.com/en-us/features/cloud-shell/) to run: `az account list-locations`\n",
                "    * To locate your tenant id, use Azure CLI to run: `az account show`"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 3,
            "metadata": {
                "tags": [
                    "parameters"
                ]
            },
            "outputs": [],
            "source": [
                "# AzureML workspace information. Set them to create a workspace.\n",
                "SUBSCRIPTION_ID = None  #'<subscription-id>'\n",
                "RESOURCE_GROUP = None   #'<resource-group>'\n",
                "WORKSPACE_NAME = None   #'<workspace-name>'\n",
                "LOCATION = None         #'<region-to-deploy-the-workspace>'\n",
                "TENANT_ID = None        #'<tenant-id>'\n",
                "\n",
                "# Remote compute (cluster) configuration. If you want to save the cost more, set these to small.\n",
                "VM_SIZE = 'STANDARD_NC6'\n",
                "VM_PRIORITY = 'lowpriority'\n",
                "# Cluster nodes\n",
                "MIN_NODES = 0\n",
                "MAX_NODES = 8\n",
                "# Hyperdrive experimentation configuration\n",
                "MAX_TOTAL_RUNS = 100  # Number of runs (training-and-evaluation) to search the best hyperparameters. \n",
                "MAX_CONCURRENT_RUNS = 8\n",
                "\n",
                "# Recommend top k items\n",
                "TOP_K = 10\n",
                "# Select MovieLens data size: 100k, 1m, 10m, or 20m\n",
                "MOVIELENS_DATA_SIZE = '100k'\n",
                "STEPS = 50000\n",
                "# Metrics to track\n",
                "RANKING_METRICS = [evaluator.ndcg_at_k.__name__, evaluator.precision_at_k.__name__]\n",
                "RATING_METRICS = [evaluator.rmse.__name__, evaluator.mae.__name__]\n",
                "PRIMARY_METRIC = evaluator.rmse.__name__\n",
                "# Data column names\n",
                "USER_COL = 'UserId'\n",
                "ITEM_COL = 'MovieId'\n",
                "RATING_COL = 'Rating'\n",
                "ITEM_FEAT_COL = 'Genres'\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 4,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "Found the config file in: /data/home/jumin/git/reco/notebooks/04_model_select_and_optimize/aml_config/config.json\n"
                    ]
                },
                {
                    "name": "stderr",
                    "output_type": "stream",
                    "text": [
                        "Falling back to use azure cli credentials. This fall back to use azure cli credentials will be removed in the next release. \n",
                        "Make sure your code doesn't require 'az login' to have happened before using azureml-sdk, except the case when you are specifying AzureCliAuthentication in azureml-sdk.\n"
                    ]
                }
            ],
            "source": [
                "if TENANT_ID:\n",
                "    auth = aml.core.authentication.InteractiveLoginAuthentication(\n",
                "        tenant_id=TENANT_ID\n",
                "    )\n",
                "else:\n",
                "    auth = None  \n",
                "\n",
                "if SUBSCRIPTION_ID and RESOURCE_GROUP and WORKSPACE_NAME and LOCATION:\n",
                "    try:\n",
                "        # Try to get existing workspace by given information\n",
                "        ws = aml.core.Workspace(\n",
                "            workspace_name=WORKSPACE_NAME,\n",
                "            subscription_id=SUBSCRIPTION_ID,\n",
                "            resource_group=RESOURCE_GROUP,\n",
                "        )\n",
                "        print(\"Found existing AzureML workspace.\")\n",
                "    except aml.exceptions.AuthenticationException:\n",
                "        # Create a new workspace\n",
                "        print(\"Creating new AzureML workspace.\")\n",
                "        ws = aml.core.Workspace.create(\n",
                "            name=WORKSPACE_NAME,\n",
                "            subscription_id=SUBSCRIPTION_ID,\n",
                "            resource_group=RESOURCE_GROUP,\n",
                "            create_resource_group=True,\n",
                "            location=LOCATION,\n",
                "            auth=auth,\n",
                "        )\n",
                "    ws.write_config()\n",
                "# If you are using an already-configured workspace config.json file\n",
                "else:\n",
                "    ws = aml.core.Workspace.from_config(auth=auth)"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "To verify your workspace, run:"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 5,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "AzureML workspace name:  junminaml\n"
                    ]
                }
            ],
            "source": [
                "print(\"AzureML workspace name: \", ws.name)"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "### 2. Create Remote Compute Target\n",
                "\n",
                "We create a GPU cluster as our **remote compute target**. If a cluster with the same name is already exist in your workspace, the script will load it instead. You can see [this document](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-set-up-training-targets) to learn more about setting up a compute target on different locations.\n",
                "\n",
                "This notebook selects **STANDARD_NC6** virtual machine (VM) and sets it's priority as *lowpriority* to save the cost.\n",
                "\n",
                "Size | vCPU | Memory (GiB) | Temp storage (SSD, GiB) | GPU | GPU memory (GiB) | Max data disks | Max NICs\n",
                "---|---|---|---|---|---|---|---\n",
                "Standard_NC6 | <div align=\"center\">6</div> | <div align=\"center\">56</div> | <div align=\"center\">340</div> | <div align=\"center\">1</div> | <div align=\"center\">8</div> | <div align=\"center\">24</div> | <div align=\"center\">1</div>\n",
                "\n",
                "\n",
                "For more information about Azure virtual machine sizes, see [here](https://docs.microsoft.com/en-us/azure/virtual-machines/windows/sizes-gpu)."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 6,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "Found existing compute target\n",
                        "{'allocationState': 'Steady', 'allocationStateTransitionTime': '2019-06-28T16:58:16.459000+00:00', 'creationTime': '2019-06-18T21:09:39.101231+00:00', 'currentNodeCount': 0, 'errors': None, 'modifiedTime': '2019-06-18T21:09:55.347615+00:00', 'nodeStateCounts': {'idleNodeCount': 0, 'leavingNodeCount': 0, 'preemptedNodeCount': 0, 'preparingNodeCount': 0, 'runningNodeCount': 0, 'unusableNodeCount': 0}, 'provisioningState': 'Succeeded', 'provisioningStateTransitionTime': None, 'scaleSettings': {'minNodeCount': 0, 'maxNodeCount': 8, 'nodeIdleTimeBeforeScaleDown': 'PT120S'}, 'targetNodeCount': 0, 'vmPriority': 'LowPriority', 'vmSize': 'STANDARD_NC6'}\n"
                    ]
                }
            ],
            "source": [
                "CLUSTER_NAME = 'gpu-cluster-nc6'\n",
                "\n",
                "try:\n",
                "    compute_target = aml.core.compute.ComputeTarget(workspace=ws, name=CLUSTER_NAME)\n",
                "    print(\"Found existing compute target\")\n",
                "except aml.core.compute_target.ComputeTargetException:\n",
                "    print(\"Creating a new compute target...\")\n",
                "    compute_config = aml.core.compute.AmlCompute.provisioning_configuration(\n",
                "        vm_size=VM_SIZE,\n",
                "        vm_priority=VM_PRIORITY,\n",
                "        min_nodes=MIN_NODES,\n",
                "        max_nodes=MAX_NODES\n",
                "    )\n",
                "    # create the cluster\n",
                "    compute_target = aml.core.compute.ComputeTarget.create(ws, CLUSTER_NAME, compute_config)\n",
                "    compute_target.wait_for_completion(show_output=True, min_node_count=None, timeout_in_minutes=20)\n",
                "\n",
                "# Use the 'status' property to get a detailed status for the current cluster. \n",
                "print(compute_target.status.serialize())"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "### 3. Prepare Data\n",
                "For demonstration purpose, we use 100k MovieLens dataset. First, download the data and convert the format (multi-hot encode *genres*) to make it work for our model. More details about this step is described in our [Wide-Deep Quickstart notebook](../00_quick_start/wide_deep_movielens.ipynb)."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 7,
            "metadata": {},
            "outputs": [
                {
                    "name": "stderr",
                    "output_type": "stream",
                    "text": [
                        "100%|██████████| 4.81k/4.81k [00:00<00:00, 18.1kKB/s]\n"
                    ]
                },
                {
                    "data": {
                        "text/html": [
                            "<div>\n",
                            "<style scoped>\n",
                            "    .dataframe tbody tr th:only-of-type {\n",
                            "        vertical-align: middle;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe tbody tr th {\n",
                            "        vertical-align: top;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe thead th {\n",
                            "        text-align: right;\n",
                            "    }\n",
                            "</style>\n",
                            "<table border=\"1\" class=\"dataframe\">\n",
                            "  <thead>\n",
                            "    <tr style=\"text-align: right;\">\n",
                            "      <th></th>\n",
                            "      <th>UserId</th>\n",
                            "      <th>MovieId</th>\n",
                            "      <th>Rating</th>\n",
                            "      <th>Genres</th>\n",
                            "    </tr>\n",
                            "  </thead>\n",
                            "  <tbody>\n",
                            "    <tr>\n",
                            "      <th>0</th>\n",
                            "      <td>196</td>\n",
                            "      <td>242</td>\n",
                            "      <td>3.0</td>\n",
                            "      <td>[0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>1</th>\n",
                            "      <td>63</td>\n",
                            "      <td>242</td>\n",
                            "      <td>3.0</td>\n",
                            "      <td>[0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>2</th>\n",
                            "      <td>226</td>\n",
                            "      <td>242</td>\n",
                            "      <td>5.0</td>\n",
                            "      <td>[0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>3</th>\n",
                            "      <td>154</td>\n",
                            "      <td>242</td>\n",
                            "      <td>3.0</td>\n",
                            "      <td>[0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>4</th>\n",
                            "      <td>306</td>\n",
                            "      <td>242</td>\n",
                            "      <td>5.0</td>\n",
                            "      <td>[0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...</td>\n",
                            "    </tr>\n",
                            "  </tbody>\n",
                            "</table>\n",
                            "</div>"
                        ],
                        "text/plain": [
                            "   UserId  MovieId  Rating                                             Genres\n",
                            "0     196      242     3.0  [0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...\n",
                            "1      63      242     3.0  [0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...\n",
                            "2     226      242     5.0  [0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...\n",
                            "3     154      242     3.0  [0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...\n",
                            "4     306      242     5.0  [0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ..."
                        ]
                    },
                    "execution_count": 7,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "data = movielens.load_pandas_df(\n",
                "    size=MOVIELENS_DATA_SIZE,\n",
                "    header=[USER_COL, ITEM_COL, RATING_COL],\n",
                "    genres_col=ITEM_FEAT_COL\n",
                ")\n",
                "\n",
                "# Encode 'genres' into int array (multi-hot representation) to use as item features\n",
                "genres_encoder = sklearn.preprocessing.MultiLabelBinarizer()\n",
                "data[ITEM_FEAT_COL] = genres_encoder.fit_transform(\n",
                "    data[ITEM_FEAT_COL].apply(lambda s: s.split(\"|\"))\n",
                ").tolist()\n",
                "\n",
                "data.head()"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "The dataset is split into train, validation, and test sets. The train and validation sets will be used for hyperparameter tuning, and the test set will be used for the final evaluation of the model after we import the best model from AzureML workspace.\n",
                "\n",
                "Here, we don't use multiple-split directly by passing `ratio=[0.56, 0.19, 0.25]`. Instead, we first split the data into train and test sets with the same `seed` we've been using in other notebooks to make the train set identical across them. Then, we further split the train set into train and validation sets."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 11,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "Number of samples:\n",
                        "- Training   = 56250\n",
                        "- Validation = 18750\n",
                        "- Testing    = 25000\n"
                    ]
                }
            ],
            "source": [
                "# Use the same seed to make the train and test sets identical across other notebooks in the repo.\n",
                "train, test = python_random_split(data, ratio=0.75, seed=SEED)\n",
                "# Further split the train set into train and validation set.\n",
                "train, valid = python_random_split(train, seed=SEED)\n",
                "\n",
                "print(\n",
                "    \"Number of samples:\\n\"\n",
                "    \"- Training   = {}\\n\"\n",
                "    \"- Validation = {}\\n\"\n",
                "    \"- Testing    = {}\".format(len(train), len(valid), len(test))\n",
                ")"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Now, upload the train and validation sets to the AzureML workspace. Our Hyperdrivce experiment will use them."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 12,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "Uploading /tmp/tmpwby7dwh4/aml_data/movielens_100k_train.pkl\n",
                        "Uploading /tmp/tmpwby7dwh4/aml_data/movielens_100k_valid.pkl\n",
                        "Uploaded /tmp/tmpwby7dwh4/aml_data/movielens_100k_valid.pkl, 1 files out of an estimated total of 2\n",
                        "Uploaded /tmp/tmpwby7dwh4/aml_data/movielens_100k_train.pkl, 2 files out of an estimated total of 2\n"
                    ]
                },
                {
                    "data": {
                        "text/plain": [
                            "$AZUREML_DATAREFERENCE_ec1d8219afb44a36adf66ff9ece918f4"
                        ]
                    },
                    "execution_count": 12,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "DATA_DIR = os.path.join(tmp_dir.name, 'aml_data') \n",
                "\n",
                "os.makedirs(DATA_DIR, exist_ok=True)\n",
                "\n",
                "TRAIN_FILE_NAME = \"movielens_\" + MOVIELENS_DATA_SIZE + \"_train.pkl\"\n",
                "train.to_pickle(os.path.join(DATA_DIR, TRAIN_FILE_NAME))\n",
                "VALID_FILE_NAME = \"movielens_\" + MOVIELENS_DATA_SIZE + \"_valid.pkl\"\n",
                "valid.to_pickle(os.path.join(DATA_DIR, VALID_FILE_NAME))\n",
                "\n",
                "# Note, all the files under DATA_DIR will be uploaded to the data store\n",
                "ds = ws.get_default_datastore()\n",
                "ds.upload(\n",
                "    src_dir=DATA_DIR,\n",
                "    target_path='data',\n",
                "    overwrite=True,\n",
                "    show_progress=True\n",
                ")"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "### 4. Prepare Training Scripts\n",
                "Next step is to prepare scripts that AzureML Hyperdrive will use to train and evaluate models with selected hyperparameters. We re-use our [Wide-Deep Quickstart notebook](../00_quick_start/wide_deep_movielens.ipynb) for that. To run the model notebook from the Hyperdrive Run, all we need is to prepare an [entry script](../../recommenders/azureml/wide_deep.py) which parses the hyperparameter arguments, passes them to the notebook, and records the results of the notebook to AzureML Run logs by using `papermill`. Hyperdrive uses the logs to track the performance of each hyperparameter-set and finds the best performed one.  \n",
                "\n",
                "Here is a code snippet from the entry script:\n",
                "```\n",
                "...\n",
                "from azureml.core import Run\n",
                "run = Run.get_context()\n",
                "...\n",
                "NOTEBOOK_NAME = os.path.join(\n",
                "    \"notebooks\",\n",
                "    \"00_quick_start\",\n",
                "    \"wide_deep_movielens.ipynb\"\n",
                ")\n",
                "...\n",
                "parser = argparse.ArgumentParser()\n",
                "...\n",
                "parser.add_argument('--dnn-optimizer', type=str, dest='dnn_optimizer', ...\n",
                "parser.add_argument('--dnn-optimizer-lr', type=float, dest='dnn_optimizer_lr', ...\n",
                "...\n",
                "pm.execute_notebook(\n",
                "    NOTEBOOK_NAME,\n",
                "    OUTPUT_NOTEBOOK,\n",
                "    parameters=params,\n",
                "    kernel_name='python3',\n",
                ")\n",
                "...\n",
                "```"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 13,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Prepare all the necessary scripts which will be loaded to our Hyperdrive Experiment Run\n",
                "SCRIPT_DIR = os.path.join(tmp_dir.name, 'aml_script')\n",
                "\n",
                "# Copy scripts to SCRIPT_DIR temporarly\n",
                "shutil.copytree(os.path.join('..', '..', 'recommenders'), os.path.join(SCRIPT_DIR, 'recommenders'))\n",
                "\n",
                "# We re-use our model notebook for training and testing models.\n",
                "model_notebook_dir = os.path.join('notebooks', '00_quick_start')\n",
                "dest_model_notebook_dir = os.path.join(SCRIPT_DIR, model_notebook_dir)\n",
                "os.makedirs(dest_model_notebook_dir , exist_ok=True)\n",
                "shutil.copy(\n",
                "    os.path.join('..', '..', model_notebook_dir, 'wide_deep_movielens.ipynb'),\n",
                "    dest_model_notebook_dir\n",
                ")\n",
                "\n",
                "# copy training scripts\n",
                "shutil.copytree('train_scripts', os.path.join(SCRIPT_DIR, 'train_scripts'))\n",
                "\n",
                "# This is our entry script for Hyperdrive Run\n",
                "ENTRY_SCRIPT_NAME = 'train_scripts/wide_deep_training.py'"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "### 5. Setup and Run Hyperdrive Experiment\n",
                "\n",
                "#### 5.1 Define Search Space \n",
                "We define the search space of hyperparameters. For example, if you want to test different batch sizes of {64, 128, 256}, you can use `azureml.train.hyperdrive.choice(64, 128, 256)`. To search from a continuous space, use `uniform(start, end)`. For more options, see [Hyperdrive parameter expressions](https://docs.microsoft.com/en-us/python/api/azureml-train-core/azureml.train.hyperdrive.parameter_expressions?view=azure-ml-py).\n",
                "In this notebook, we fix the number of training steps to 50000.\n",
                "\n",
                "In the search space, we set different linear and DNN optimizers, structures, learning rates and regularization rates. Details about the hyperparameters can be found from our [Wide-Deep Quickstart notebook](../00_quick_start/wide_deep_movielens.ipynb)."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 14,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Script parameters. New AzureML API only accepts string values.\n",
                "script_params = {\n",
                "    '--datastore': ds.as_mount(),\n",
                "    '--train-datapath': \"data/\" + TRAIN_FILE_NAME,\n",
                "    '--test-datapath': \"data/\" + VALID_FILE_NAME,\n",
                "    '--top-k': str(TOP_K),\n",
                "    '--user-col': USER_COL,\n",
                "    '--item-col': ITEM_COL,\n",
                "    '--item-feat-col': ITEM_FEAT_COL,\n",
                "    '--rating-col': RATING_COL,\n",
                "    '--ranking-metrics': RANKING_METRICS,\n",
                "    '--rating-metrics': RATING_METRICS,\n",
                "    '--steps': str(STEPS),\n",
                "}\n",
                "\n",
                "# Hyperparameter search space\n",
                "params = {\n",
                "    '--model-type': hd.choice('wide', 'deep', 'wide_deep'),\n",
                "    '--batch-size': hd.choice(8, 16, 32, 64, 128),\n",
                "    # Linear model hyperparameters\n",
                "    '--linear-optimizer': hd.choice('adadelta', 'adagrad', 'adam', 'ftrl', 'momentum', 'sgd'),\n",
                "    '--linear-optimizer-lr': hd.uniform(1e-6, 0.1),\n",
                "    '--linear-l1-reg': hd.uniform(0.0, 1.0),\n",
                "    '--linear-l2-reg': hd.uniform(0.0, 1.0),\n",
                "    '--linear-momentum': hd.uniform(0.0, 1.0),\n",
                "    # Deep model hyperparameters\n",
                "    '--dnn-optimizer': hd.choice('adadelta', 'adagrad', 'adam', 'ftrl', 'momentum', 'sgd'),\n",
                "    '--dnn-optimizer-lr': hd.uniform(1e-6, 0.1),\n",
                "    '--dnn-l1-reg': hd.uniform(0.0, 1.0),\n",
                "    '--dnn-l2-reg': hd.uniform(0.0, 1.0),\n",
                "    '--dnn-momentum': hd.uniform(0.0, 1.0),\n",
                "    '--dnn-user-embedding-dim': hd.choice(4, 8, 16, 32),\n",
                "    '--dnn-item-embedding-dim': hd.choice(4, 8, 16, 32),\n",
                "    '--dnn-hidden-layer-1': hd.choice(0, 64, 128, 256, 512),  # 0: not using this layer\n",
                "    '--dnn-hidden-layer-2': hd.choice(0, 64, 128, 256, 512),\n",
                "    '--dnn-hidden-layer-3': hd.choice(0, 64, 128, 256, 512),\n",
                "    '--dnn-hidden-layer-4': hd.choice(64, 128, 256, 512, 1024),\n",
                "    '--dnn-batch-norm': hd.choice(0, 1),\n",
                "    '--dnn-dropout': hd.uniform(0.0, 0.8)\n",
                "}"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "\n",
                "#### 5.2 Create Hyperdrive Experiment \n",
                "[Hyperdrive](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-tune-hyperparameters) creates a machine learning experiment [**Run**](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.run?view=azure-ml-py) on the workspace and utilizes child-runs to search the best set of hyperparameters. [Experiment](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.experiment(class)?view=azure-ml-py) is the main entry point into experimenting with AzureML. To create new Experiment or get the existing one, we pass our experimentation name.\n",
                "\n",
                "**AzureML Estimator** is the building block for training. An Estimator encapsulates the training code and parameters, the compute resources and runtime environment for a particular training scenario (Note, this is not TensorFlow's Estimator). In the following cell, we create the Estimator with additional dependencies of our model scripts."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 15,
            "metadata": {},
            "outputs": [],
            "source": [
                "est = aml.train.estimator.Estimator(\n",
                "    source_directory=SCRIPT_DIR,\n",
                "    entry_script=ENTRY_SCRIPT_NAME,\n",
                "    script_params=script_params,\n",
                "    compute_target=compute_target,\n",
                "    use_gpu=True,\n",
                "    conda_packages=['pandas', 'scikit-learn', 'numba', 'matplotlib'],\n",
                "    pip_packages=['ipykernel', 'papermill==0.18.2', 'tensorflow-gpu==1.12']\n",
                ")"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "We set our primary metric with the goal (hyperparameter search criteria), hyperparameter sampling method, and number of total child-runs to the Hyperdrive Run Config. The bigger the search space, the more number of runs we will need for better results.\n",
                "\n",
                "Hyperdrive provides three different parameter sampling methods: `RandomParameterSampling`, `GridParameterSampling`, and `BayesianParameterSampling`. Details about each method can be found from [Azure doc](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-tune-hyperparameters). Here, we use the Bayesian sampling."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 16,
            "metadata": {},
            "outputs": [],
            "source": [
                "hd_run_config = hd.HyperDriveRunConfig(\n",
                "    estimator=est, \n",
                "    hyperparameter_sampling=hd.BayesianParameterSampling(params),\n",
                "    primary_metric_name=PRIMARY_METRIC,\n",
                "    primary_metric_goal=hd.PrimaryMetricGoal.MINIMIZE, \n",
                "    max_total_runs=MAX_TOTAL_RUNS,\n",
                "    max_concurrent_runs=MAX_CONCURRENT_RUNS\n",
                ")"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "#### 5.3 Run Experiment\n",
                "\n",
                "Now we submit the Run to our experiment. You can see the experiment progress from this notebook by using `azureml.widgets.RunDetails(hd_run).show()` or check from the Azure portal with the url link you can get by running `hd_run.get_portal_url()`.\n",
                "\n",
                "<img src=\"https://recodatasets.z20.web.core.windows.net/images/aml_0.png?sanitize=true\" width=\"600\"/>\n",
                "<img src=\"https://recodatasets.z20.web.core.windows.net/images/aml_1.png?sanitize=true\" width=\"600\"/>\n",
                "<center><i>AzureML Hyperdrive Widget</i></center>\n",
                "\n",
                "To load an existing Hyperdrive Run instead of start new one, use `hd_run = hd.HyperDriveRun(exp, <user-run-id>, hyperdrive_run_config=hd_run_config)`. You also can cancel the Run with `hd_run.cancel()`."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 17,
            "metadata": {},
            "outputs": [],
            "source": [
                "EXP_NAME = \"movielens_\" + MOVIELENS_DATA_SIZE + \"_wide_deep_model\"\n",
                "exp = aml.core.Experiment(workspace=ws, name=EXP_NAME)"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Create an experiment run. Skip this to load an existing run instead\n",
                "hd_run = exp.submit(config=hd_run_config)\n",
                "\n",
                "# To load an existing run: \n",
                "# hd_run = hd.HyperDriveRun(\n",
                "#     experiment=exp,\n",
                "#     run_id=<run-id-to-load>,\n",
                "#     hyperdrive_run_config=hd_run_config\n",
                "# )\n",
                "\n",
                "hd_run.get_details()"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 20,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/plain": [
                            "[Run(Experiment: movielens_100k_wide_deep_model,\n",
                            " Id: movielens_100k_wide_deep_model_1561733572398,\n",
                            " Type: hyperdrive,\n",
                            " Status: Completed), Run(Experiment: movielens_100k_wide_deep_model,\n",
                            " Id: movielens_100k_wide_deep_model_1561703444608,\n",
                            " Type: hyperdrive,\n",
                            " Status: Canceled), Run(Experiment: movielens_100k_wide_deep_model,\n",
                            " Id: movielens_100k_wide_deep_model_1560996258088,\n",
                            " Type: hyperdrive,\n",
                            " Status: Completed), Run(Experiment: movielens_100k_wide_deep_model,\n",
                            " Id: movielens_100k_wide_deep_model_1560994940938,\n",
                            " Type: hyperdrive,\n",
                            " Status: Canceled), Run(Experiment: movielens_100k_wide_deep_model,\n",
                            " Id: movielens_100k_wide_deep_model_1560993611286,\n",
                            " Type: hyperdrive,\n",
                            " Status: Canceled), Run(Experiment: movielens_100k_wide_deep_model,\n",
                            " Id: movielens_100k_wide_deep_model_1560892511300,\n",
                            " Type: hyperdrive,\n",
                            " Status: Canceled), Run(Experiment: movielens_100k_wide_deep_model,\n",
                            " Id: movielens_100k_wide_deep_model_1562013419918,\n",
                            " Type: hyperdrive,\n",
                            " Status: Running)]"
                        ]
                    },
                    "execution_count": 20,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "# Get the list of runs from the experiment:\n",
                "list(exp.get_runs())"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Note, widgets don't work on JupyterLab\n",
                "widgets.RunDetails(hd_run).show()"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Once all the child-runs are finished, we can get the best run and the metrics.\n",
                "> Note, if you run Hyperdrive experiment again, you will see the best metrics and corresponding hyperparameters are not the same. It is because of 1) the random initialization of the model and 2) Hyperdrive sampling (when you use RandomSampling). You will get different results as well if you use different training and validation sets."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 17,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Get best run and printout metrics\n",
                "best_run = hd_run.get_best_run_by_primary_metric()\n",
                "best_run_metrics = best_run.get_metrics()"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 18,
            "metadata": {
                "scrolled": false
            },
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "* Best Run Id: movielens_100k_wide_deep_model_1561733572398_41\n",
                        "\n",
                        "* Best hyperparameters:\n",
                        "Model type = wide_deep\n",
                        "Batch size = 32.0\n",
                        "Linear optimizer = adagrad\n",
                        "\tLearning rate = 0.0621\n",
                        "DNN optimizer = adadelta\n",
                        "\tUser embedding dimension = 32.0\n",
                        "\tItem embedding dimension = 16.0\n",
                        "\tHidden units = [0.0, 64.0, 128.0, 512.0]\n",
                        "\tLearning rate = 0.1000\n",
                        "\tDropout rate = 0.8000\n",
                        "\tBatch normalization = True\n",
                        "\n",
                        "* Performance metrics:\n",
                        "\tndcg_at_k (top-10) = 0.0555\n",
                        "\tprecision_at_k (top-10) = 0.0534\n",
                        "\trmse = 0.9552\n",
                        "\tmae = 0.7568\n"
                    ]
                }
            ],
            "source": [
                "print(\"* Best Run Id:\", best_run.id)\n",
                "\n",
                "print(\"\\n* Best hyperparameters:\")\n",
                "model_type = best_run_metrics['MODEL_TYPE']\n",
                "print(\"Model type =\", model_type)\n",
                "print(\"Batch size =\", best_run_metrics['BATCH_SIZE'])\n",
                "\n",
                "if model_type in ('wide', 'wide_deep'):\n",
                "    linear_opt = best_run_metrics['LINEAR_OPTIMIZER']\n",
                "    print(\"Linear optimizer =\", linear_opt)\n",
                "    print(\"\\tLearning rate = {0:.4f}\".format(best_run_metrics['LINEAR_OPTIMIZER_LR']))\n",
                "    if linear_opt == 'ftrl':\n",
                "        print(\"\\tL1 regularization = {0:.4f}\".format(best_run_metrics['LINEAR_L1_REG']))\n",
                "        print(\"\\tL2 regularization = {0:.4f}\".format(best_run_metrics['LINEAR_L2_REG']))\n",
                "    elif linear_opt == 'momentum' or linear_opt == 'rmsprop':\n",
                "        print(\"\\tMomentum = {0:.4f}\".format(best_run_metrics['LINEAR_MOMENTUM']))\n",
                "\n",
                "if model_type in ('deep', 'wide_deep'):\n",
                "    dnn_opt = best_run_metrics['DNN_OPTIMIZER']\n",
                "    print(\"DNN optimizer =\", dnn_opt)\n",
                "    print(\"\\tUser embedding dimension =\", best_run_metrics['DNN_USER_DIM'])\n",
                "    print(\"\\tItem embedding dimension =\", best_run_metrics['DNN_ITEM_DIM'])\n",
                "    print(\"\\tHidden units =\", [\n",
                "        best_run_metrics['DNN_HIDDEN_LAYER_{}'.format(i)] for i in range(1, 5)\n",
                "    ])\n",
                "    print(\"\\tLearning rate = {0:.4f}\".format(best_run_metrics['DNN_OPTIMIZER_LR']))\n",
                "    if dnn_opt == 'ftrl':\n",
                "        print(\"\\tL1 regularization = {0:.4f}\".format(best_run_metrics['DNN_L1_REG']))\n",
                "        print(\"\\tL2 regularization = {0:.4f}\".format(best_run_metrics['DNN_L2_REG']))\n",
                "    elif dnn_opt == 'momentum' or linear_opt == 'rmsprop':\n",
                "        print(\"\\tMomentum = {0:.4f}\".format(best_run_metrics['DNN_MOMENTUM']))\n",
                "    print(\"\\tDropout rate = {0:.4f}\".format(best_run_metrics['DNN_DROPOUT']))\n",
                "    print(\"\\tBatch normalization =\", 1==best_run_metrics['DNN_BATCH_NORM'])\n",
                "    \n",
                "# Metrics evaluated on validation set\n",
                "print(\"\\n* Performance metrics:\")\n",
                "for m in RANKING_METRICS:\n",
                "    print(\"\\t{0} (top-{1}) = {2:.4f}\".format(m, TOP_K, best_run_metrics[m]))\n",
                "for m in RATING_METRICS:\n",
                "    print(\"\\t{0} = {1:.4f}\".format(m, best_run_metrics[m]))    "
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "### 6. Model Import and Test\n",
                "\n",
                "[Wide-Deep Quickstart notebook](../00_quick_start/wide_deep_movielens.ipynb), which we've used in our Hyperdrive Experiment, exports the trained model to the output folder (the output path is recorded at `best_run_metrics['saved_model_dir']`). We can download a model from the best run and test it. "
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 43,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "outputs/model/1561737321/\n",
                        "Downloading outputs/model/1561737321/saved_model.pb..\n",
                        "Downloading outputs/model/1561737321/variables/variables.data-00000-of-00002..\n",
                        "Downloading outputs/model/1561737321/variables/variables.data-00001-of-00002..\n",
                        "Downloading outputs/model/1561737321/variables/variables.index..\n"
                    ]
                }
            ],
            "source": [
                "MODEL_DIR = os.path.join(tmp_dir.name, 'aml_model')\n",
                "os.makedirs(MODEL_DIR, exist_ok=True)\n",
                "\n",
                "model_file_dir = best_run_metrics['saved_model_dir'] + '/'\n",
                "print(model_file_dir)\n",
                "\n",
                "for f in best_run.get_file_names():\n",
                "    if f.startswith(model_file_dir):\n",
                "        output_file_path = os.path.join(MODEL_DIR, f.split(model_file_dir)[1])\n",
                "        print(\"Downloading {}..\".format(f))\n",
                "        best_run.download_file(name=f, output_file_path=output_file_path)\n",
                "    \n",
                "saved_model = tf.saved_model.load(MODEL_DIR, tags=\"serve\")"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 44,
            "metadata": {},
            "outputs": [],
            "source": [
                "cols = {\n",
                "    'col_user': USER_COL,\n",
                "    'col_item': ITEM_COL,\n",
                "    'col_rating': RATING_COL,\n",
                "    'col_prediction': 'prediction'\n",
                "}\n",
                "\n",
                "tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.ERROR)"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 46,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "count    25000.000000\n",
                        "mean         3.525522\n",
                        "std          0.635910\n",
                        "min          0.140751\n",
                        "25%          3.129608\n",
                        "50%          3.576132\n",
                        "75%          3.973043\n",
                        "max          5.629328\n",
                        "Name: prediction, dtype: float64 \n",
                        "\n",
                        "rmse = 0.956280219325999\n",
                        "mae = 0.7553600390541554\n"
                    ]
                }
            ],
            "source": [
                "# Rating prediction set\n",
                "X_test = test.drop(RATING_COL, axis=1)\n",
                "X_test.reset_index(drop=True, inplace=True)\n",
                "\n",
                "# Rating prediction\n",
                "predictions = list(itertools.islice(\n",
                "    saved_model.signatures[\"predict\"](\n",
                "        examples=pandas_input_fn_for_saved_model(\n",
                "            df=X_test,\n",
                "            feat_name_type={\n",
                "                USER_COL: int,\n",
                "                ITEM_COL: int,\n",
                "                ITEM_FEAT_COL: list\n",
                "            }\n",
                "        )()[\"inputs\"]\n",
                "    ),\n",
                "    len(X_test)\n",
                "))\n",
                "\n",
                "prediction_df = X_test.copy()\n",
                "prediction_df['prediction'] = [p['outputs'][0] for p in predictions]\n",
                "print(prediction_df['prediction'].describe(), \"\\n\")\n",
                "for m in RATING_METRICS:\n",
                "    result = evaluator.metrics[m](test, prediction_df, **cols)\n",
                "    print(m, \"=\", result)"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 32,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Unique items\n",
                "if ITEM_FEAT_COL is None:\n",
                "    items = data.drop_duplicates(ITEM_COL)[[ITEM_COL]].reset_index(drop=True)\n",
                "else:\n",
                "    items = data.drop_duplicates(ITEM_COL)[[ITEM_COL, ITEM_FEAT_COL]].reset_index(drop=True)\n",
                "# Unique users\n",
                "users = data.drop_duplicates(USER_COL)[[USER_COL]].reset_index(drop=True)\n",
                "\n",
                "# Ranking prediction set\n",
                "ranking_pool = user_item_pairs(\n",
                "    user_df=users,\n",
                "    item_df=items,\n",
                "    user_col=USER_COL,\n",
                "    item_col=ITEM_COL,\n",
                "    user_item_filter_df=pd.concat([train, valid]),  # remove seen items\n",
                "    shuffle=True\n",
                ")"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 33,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "ndcg_at_k = 0.018009288572177713\n",
                        "precision_at_k = 0.01792152704135737\n"
                    ]
                }
            ],
            "source": [
                "predictions = []\n",
                "# If we put all ranking_pool into a tensor, we get error (since the content limit is 2GB).\n",
                "# We divide ranking_pool into 5 chunks, make prediction, and concat the results. \n",
                "for pool in np.array_split(ranking_pool, 5):\n",
                "    pool.reset_index(drop=True, inplace=True)\n",
                "    # Rating prediction\n",
                "    pred = list(itertools.islice(\n",
                "        saved_model.signatures[\"predict\"](\n",
                "            examples=pandas_input_fn_for_saved_model(\n",
                "                df=X_test,\n",
                "                feat_name_type={\n",
                "                    USER_COL: int,\n",
                "                    ITEM_COL: int,\n",
                "                    ITEM_FEAT_COL: list\n",
                "                }\n",
                "            )()[\"inputs\"]\n",
                "        ),\n",
                "        len(pool)\n",
                "    ))\n",
                "    predictions.extend([p['outputs'][0] for p in pred])\n",
                "    \n",
                "ranking_pool['prediction'] = predictions\n",
                "\n",
                "for m in RANKING_METRICS:\n",
                "    result = evaluator.metrics[m](test, ranking_pool, **{**cols, 'k': TOP_K})\n",
                "    print(m, \"=\", result)"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "#### Wide-and-Deep Baseline Comparison\n",
                "To see if Hyperdrive found good hyperparameters, we simply compare with the model with known hyperparameters from [TensorFlow's wide-deep learning example](https://github.com/tensorflow/models/blob/master/official/wide_deep/movielens_main.py) which uses only the DNN part from the wide-and-deep model for MovieLens data.\n",
                "\n",
                "> Note, this is not 'apples to apples' comparison. For example, TensorFlow's movielens example uses *rating-timestamp* as a numeric feature, but we did not use that here because we think the timestamps are not relevant to the movies' ratings. This comparison is more like to show how Hyperdrive can help to find comparable hyperparameters without requiring exhaustive efforts in going over a huge search-space. "
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 47,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "application/vnd.jupyter.widget-view+json": {
                            "model_id": "ec87f42770694ffcba842b3fa8e93e2f",
                            "version_major": 2,
                            "version_minor": 0
                        },
                        "text/plain": [
                            "HBox(children=(IntProgress(value=0, max=34), HTML(value='')))"
                        ]
                    },
                    "metadata": {},
                    "output_type": "display_data"
                },
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "\n",
                        "Training and evaluation of Wide-and-Deep model took 357.3825697898865 secs.\n",
                        "ndcg_at_k = 0.013269362558705873\n",
                        "precision_at_k = 0.015482502651113467\n",
                        "rmse = 1.0421873135289017\n",
                        "mae = 0.8238318599748612\n"
                    ]
                },
                {
                    "name": "stderr",
                    "output_type": "stream",
                    "text": [
                        "/data/anaconda/envs/reco_gpu/lib/python3.6/site-packages/ipykernel_launcher.py:37: DeprecationWarning: Function read_notebook is deprecated and will be removed in verison 1.0.0 (current version 0.19.0). Please see `scrapbook.read_notebook` (nteract-scrapbook) as a replacement for this functionality.\n"
                    ]
                }
            ],
            "source": [
                "OUTPUT_NOTEBOOK = os.path.join(tmp_dir.name, \"output.ipynb\")\n",
                "OUTPUT_MODEL_DIR = os.path.join(tmp_dir.name, \"known_hyperparam_model_checkpoints\")\n",
                "\n",
                "params = {\n",
                "    'MOVIELENS_DATA_SIZE': MOVIELENS_DATA_SIZE,\n",
                "    'TOP_K': TOP_K,\n",
                "    'MODEL_TYPE': 'deep',\n",
                "    'STEPS': STEPS,\n",
                "    'BATCH_SIZE': 256,\n",
                "    'DNN_OPTIMIZER': 'Adam',\n",
                "    'DNN_OPTIMIZER_LR': 0.001,\n",
                "    'DNN_HIDDEN_LAYER_1': 256,\n",
                "    'DNN_HIDDEN_LAYER_2': 256,\n",
                "    'DNN_HIDDEN_LAYER_3': 256,\n",
                "    'DNN_HIDDEN_LAYER_4': 128,\n",
                "    'DNN_USER_DIM': 16,\n",
                "    'DNN_ITEM_DIM': 64,\n",
                "    'DNN_DROPOUT': 0.3,\n",
                "    'DNN_BATCH_NORM': 0,\n",
                "    'MODEL_DIR': OUTPUT_MODEL_DIR,\n",
                "    'EVALUATE_WHILE_TRAINING': False,\n",
                "    'EXPORT_DIR_BASE': OUTPUT_MODEL_DIR,\n",
                "    'RANKING_METRICS': RANKING_METRICS,\n",
                "    'RATING_METRICS': RATING_METRICS,\n",
                "}\n",
                "\n",
                "with Timer() as train_time:\n",
                "    pm.execute_notebook(\n",
                "        \"../00_quick_start/wide_deep_movielens.ipynb\",\n",
                "        OUTPUT_NOTEBOOK,\n",
                "        parameters=params,\n",
                "        kernel_name='python3'\n",
                "    )\n",
                "print(\"Training and evaluation of Wide-and-Deep model took\", train_time.interval, \"secs.\")\n",
                "\n",
                "nb = pm.read_notebook(OUTPUT_NOTEBOOK)\n",
                "for m in RANKING_METRICS:\n",
                "    print(m, \"=\", nb.data[m])\n",
                "for m in RATING_METRICS:\n",
                "    print(m, \"=\", nb.data[m])"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "### Concluding Remark\n",
                "We showed how to tune hyperparameters by utilizing Azure Machine Learning service. Complex and powerful models like Wide-and-Deep model often have many number of hyperparameters that affect on the recommendation accuracy, and it is not practical to tune the model without using a GPU cluster. For example, a training and evaluation of a model took around 3 minutes on 100k MovieLens data on a single *Standard NC6* VM as we tested from the [above cell](#Wide-and-Deep-Baseline-Comparison). When we used 1M MovieLens, it took about 47 minutes. If we want to investigate through 100 different combinations of hyperparameters **manually**, it will take **78 hours** on the VM and we may still wonder if we had tested good candidates of hyperparameters. With AzureML, as we shown in this notebook, we can easily setup different size of GPU cluster fits to our problem and utilize Bayesian sampling to navigate through the huge search space efficiently, and tweak the experiment with different criteria and algorithms for further research."
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "#### Cleanup"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 4,
            "metadata": {},
            "outputs": [],
            "source": [
                "tmp_dir.cleanup()"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": []
        }
    ],
    "metadata": {
        "celltoolbar": "Tags",
        "kernelspec": {
            "display_name": "Python (reco_gpu)",
            "language": "python",
            "name": "reco_gpu"
        },
        "language_info": {
            "codemirror_mode": {
                "name": "ipython",
                "version": 3
            },
            "file_extension": ".py",
            "mimetype": "text/x-python",
            "name": "python",
            "nbconvert_exporter": "python",
            "pygments_lexer": "ipython3",
            "version": "3.6.7"
        }
    },
    "nbformat": 4,
    "nbformat_minor": 2
}
